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LONG-TERM  GOALS 

Our  long-tenn  goal  is  to  implement  an  operational  high-resolution  atmospheric  data  assimilation 
system  and  to  use  it  for  daily  weather  forecasting.  To  date  we  have  worked  on  several  operational  and 
scientific  aspects  of  the  problem  using  the  Coupled  Ocean/ Atmosphere  Mesoscale  Prediction  System 
(COAMPS®1):  (a)  setting  up  stable  large  scale  data  transfer  capabilities  to  facilitate  COAMPS  runs  2- 
4  times  per  day  in  an  operational  manner  for  different  geographical  regions  and  using  different  nested 
grid  configurations;  (b)  perform  research  on  the  MPI  scalability  of  COAMPS  on  selected  computer 
architectures  and  to  optimize  the  code  to  take  advantage  of  the  vector  capabilities  of  the  Cray  XI  and 
the  massively  parallel  features  of  our  Linux  cluster;  (c)  identify  and  understand  the  uncertainties  in 
high-resolution  NWP  forecast  and  their  impact  on  severe  weather,  such  as  extreme  rainfall,  and  to 
develop  model  metrics  appropriate  to  mesoscale,  detailed  weather  forecasts;  (d)  improve  our 
knowledge  of  observational  error  characteristics  for  spatially  correlated  data  and  develop  the  numerical 
schemes  capable  of  assimilating  these  types  of  observations. 

OBJECTIVES 

The  objectives  of  this  project  are  to:  (a)  implement  an  operational  data  feed  from  the  Navy  Operational 
Global  Atmospheric  Prediction  System  (NOGAPS),  and  implement  a  semi-operational  version  of 
COAMPS  at  the  Interdisciplinary  Centre  for  Mathematical  and  Computational  Modelling  (ICM), 
Warsaw  University;  (b)  validate  COAMPS  model  performance  through  inter-comparisons  with 
statistics  obtained  from  the  United  Kingdom  Meteorological  Office  (UKMO)  unified  model;  (c) 
investigate  the  scalability  of  COAMPS  on  ICM  computers,  a  200-processor  OPTERON  cluster  and  a 
Cray  XI;  and  (d)  develop  a  data  assimilation  scheme  that  can  assimilate  remotely-sensed  and  non- 
conventional  data  sources  with  a  special  emphasis  on  Doppler  radar  data.  Meeting  these  objectives 
will  allow  the  Polish  National  Air  Defense  to  issue  1-5  day  mesoscale  weather  forecasts  in  the  regions 
of  their  interest,  including  Poland  and  central  Europe. 


1  CO  AMPS®  is  a  registered  trademark  of  the  Naval  Research  Laboratory. 
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APPROACH 


Our  approach  is  to  utilize  NOGAPS  for  initial  and  lateral  boundary  conditions,  and  COAMSP  for 
mesoscale  atmospheric  forecasts.  The  NOGAPS  fields  are  obtained  from  the  Global  Ocean  Data 
Assimilation  Experiment  (GODAE)  server  at  the  Fleet  Numerical  Meteorology  and  Oceanography 
Center  (FNMOC)  and  transferred  to  ICM  in  automated  (machine-controlled)  efficient  and  stable  way 
thanks  to  support  of  FNMOC.  We  will  port  the  COAMPS  system  to  the  ICM  Linux  cluster  and  Cray 
XI  computers  and  measure  system  perfonnance  and  scalability  using  tools  developed  in-house  for 
model  verification.  The  unique  aspect  of  our  capabilities  is  that  we  concurrently  run  UKMO  mesoscale 
model  on  the  grid  which  is  similar  to  one  that  of  COAMPS.  We  will  also  investigate  the  time 
evolution  of  the  conditional  forecast  (background)  error  probability  density  function  using  an  ensemble 
of  the  model  forecast  to  generate  background  error  statistics.  This  helps  us  to  identify  and  understand 
the  uncertainties  in  high-resolution  NWP  forecast  on  high  impact  weather,  particularly  extreme 
rainfall.  Finally,  we  will  study  observational  error  characteristics  of  radar  reflectivity  and  radar  radial 
winds.  Such  observations  have  the  potential  to  provide  detailed  information  to  improve  mesoscale 
analyses  and  forecast.  We  will  study  historical  weather  events  for  which  we  have  radar  data  to 
understand  the  observational  error  characteristics.  We  will  investigate  how  they  may  be  applied  in 
data  analyses  used  for  assimilating  radar  data  into  numerical  weather  prediction  models. 

WORK  COMPLETED 

During  FY05  we  accomplished  the  following  tasks:  (a)  installed,  tested,  and  tuned  the  US  NAVY 
COAMPS  system  on  ICM  machines  at  Warsaw  University;  (b)  established  and  developed  a  stable  data 
transfer  capability  so  that  we  now  run  the  COAMPS  system  two  times  a  day  in  an  operational  manner, 
(c)  developed  tools  for  graphical  model  display,  (d)  collected  radar  data  for  post  analysis,  (e) 
developed  stable  numerical  methods  for  advection-diffusion  problems  (Witek  et  al.,  2005),  and  (f) 
implemented  and  developed  tools  to  assimilate  radar  reflectivity  and  radial  wind  data  using  the 
ensemble  Kalman  filter  approach  (Jakubiak,  2005)  in  COAMPS. 

RESULTS 

During  the  first  year  of  this  project,  we  implemented  the  CO  AMPS  atmospheric  model  on  several 
computer  architectures,  investigated  model  scalability  including  MPI,  established  operational 
procedures,  wrote  a  graphical  interface,  and  initiated  large-scale  data  transfer  from  the  GODAE  data 
server.  The  model  was  tuned  to  the  CRAY  XI  environment.  The  MPI  communication  was  tuned 
mainly  by  the  proper  selection  of  the  buffer  length.  Table  1  shows  wall  clock  time  of  the  forecast  for 
two  grids  of  moderate  sizes  (193x121x30  grid  points,  60h  and  169x217x30  grid  points,  48h). 
Comparison  of  results  on  two  different  machines  (CRAY  XI  and  Opteron  cluster)  is  very  revealing. 
Using  the  Cray  XI  with  32  single-streaming  processors,  we  required  1  hour  computer  time  for  a  60 
hour  forecast.  On  the  other  hand,  using  our  Linux  cluster  with  32  Opteron  processors  the  same 
forecast  is  requires  only  two  times  more  computer  time.  These  results  are  very  encouraging  for  the 
operational  set  up  of  the  COAMPS  system. 
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Table  1.  Comparison  of  execution  times  on  Cray  XI  and  Opteron  cluster. 


2  proc 

4  proc 

9  proc 

16  proc 

32  proc 

Cray  XI 

30426  s 

6278  s 

3803  s 

Opteron  cluster 

53208  s 

26625  s 

12368  s 

10923  s 

6888  s 

From  the  beginning  of  2005,  we  have  been  testing  different  aspects  of  the  operational  set  up  of  the 
system.  We  developed  scripting  software  to  control  data  transfer  from  the  GODAE  server.  We  are 
using  NOGAPS  global  model  results  to  specify  lateral  boundary  conditions  for  all  COAMPS  runs. 
Also,  we  use  NOGAPS  fields  to  produce  the  initial  conditions  for  all  COAMPS  runs.  Our  system 
employs  five  grids.  The  outer  grid,  with  the  spatial  resolution  of  39  km,  covers  all  of  Europe  and  the 
western  portion  of  Asia.  Within  this  grid,  we  defined  three  grids  with  resolutions  of  13  km  that  cover 
Central  Europe,  Iraq,  and  Afghanistan  (there  are  plans  to  use  COAMPS  to  support  deployment  Polish 
military  in  these  regions).  The  fifth  grid  is  centered  over  Poland  and  has  a  spatial  resolution  of  4.3  km. 
Fig.  1,  which  is  also  our  main  www  page  for  COAMPS  forecasts,  shows  the  geographical  positions  of 
these  grids. 

At  present,  we  run  the  model  two  times  a  day.  ft  takes  about  3.8  hours  on  the  Cray  XI  to  get  the  final 
120h  forecast  for  the  coarse  grid,  the  72h  forecasts  on  all  of  the  13  km  grids,  and  the  24h  forecast  for 
the  4.3  km  grid  over  Poland.  The  processing  of  data  before  the  forecast  (decoding,  checking,  and 
converting  to  the  COAMPS  flat  files)  and  post-processing  of  results  is  done  on  a  dedicated  machine. 
The  selected  results  are  converted  to  a  graphical  fonn  and  presented  on  our  internal  WEB  page.  At  the 
present  time,  the  results  of  the  COAMPS  model  runs  at  1CM  are  not  available  for  public  use,  as  more 
validation  and  verification  of  results  is  needed  to  assure  the  quality  of  our  forecasts. 

Fig.  2  shows  the  air  temperature  field  at  the  2  m  level  on  our  finest,  4.3  km  grid.  Colder  mountainous 
areas  and  different  circulations  in  the  northern  and  southern  parts  of  Poland  are  the  most  interesting 
features  on  this  picture. 
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COAMPS 


start  prognozy  t  19.9.2005  godziua  0 


Figure  1  Operational  configuration  of  the  COAMPS  system  used  in  ICM. 
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Fig.  2  Example  of  the  CO  AMPS  forecast  for  the  finer  (4.3  km)  grid.  This  is  an  18h  forecast  of  air 
temperature  and  stream  lines  at  the  2  in  level.  Starting  time:  00Z 19  Sep  2005. 
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Fig.  3  Example  of  the  COAMPS  forecast  for  the  outer,  coarse  (39  km)  grid.  This  is  a  42h  300  hPa 

wind  forecast.  Starting  date:  12Z  25  Sep  2005. 


Fig.  4  Example  of  the  COAMPS  forecast  for  the  outer,  coarse  (39  km)  grid.  This  is  a  120h  300  hPa 

wind  forecast.  Starting  date:  12Z  25  Sep  2005. 
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Figures  3  and  4  show  the  wind  fields  at  the  300  hPa  level.  Fig.  3  presents  the  42  hour  forecast.  Two  jet 
streams  are  visible,  one  over  England  and  Scandinavia,  and  the  second  in  Southern  Europe.  Fig.  4 
shows  a  120  hour  forecast. 

High-resolution  model  runs  need  new  types  of  data  to  describe  detailed  features  of  the  initial  flow. 
Recently  we  began  working  on  the  assimilation  of  radar  reflectivity  and  radial  wind  data  in  CO  AMPS. 
Our  approach  is  to  apply  the  ensemble  Kalman  filter  scheme.  In  our  preliminary  tests,  we  performed 
the  “twin-experiments”  study,  using  the  radar  reflectivity  and  radial  wind  data  created  during  a 
reference  model  run  as  the  observations.  These  extracted  “measurements”  were  assimilated  next  in  a 
control  run  of  the  same  model.  In  the  forecast  step  of  the  Kalman  filter  we  used  the  COAMPS  model. 
In  the  analysis  step  of  the  filter,  we  applied  software  developed  in  NERC,  Norway  by  Evensen  (2003) 
for  oceanographic  applications.  Details  of  these  experiments  will  be  presented  on  1 1th  Conference  on 
Mesoscale  Processes  (Jakubiak,  2005). 

In  order  to  investigate  the  observational  error  characteristics  for  spatially  correlated  data  we  collected 
radar  data  in  area  around  the  Baltic  Sea  basin.  These  data  consist  of  22  months  of  radar  composites 
with  primary  radar  data  every  15  minutes,  and  estimated  precipitation  patterns  every  3  hours. 

PERSONNEL  EXCHANGES  AND  TRAVEL  COMPLEMENTED 

Richard  Hodur,  NRL  -  visited  University  of  Warsaw  (Poland),  to  present  recent  developments  in 
COAMPS  system  and  to  discuss  proposed  set  up  of  the  COAMPS  system  at  ICM.  His  experience  in 
all  aspects  of  the  numerical  weather  prediction  modeling  was  beneficial  for  the  successful 
implementation  of  the  COAMPS  system  in  Poland. 

Marcin  Witek,  University  of  Warsaw  -  visited  NATO  Undersea  Research  Centre  (NURC,  La 
Spezia,  Italy)  to  work  with  Joao  Teixeira  on  the  preparation  of  the  paper  on  stable  and  explicit 
diffusion  and  advection-diffusion  numerical  methods.  We  look  at  his  work  as  one  of  the  possible  tools 
to  improve  the  description  of  the  advection-diffusion  of  aerosols  in  COAMPS. 

Piotr  Flatau,  NRL  -  visited  University  of  Warsaw  (Poland)  to  discuss  selected  aspects  of  our  NICOP 
project  and  to  advise  appropriate  solutions  (tools  for  model  verification,  interaction  with  related 
projects). 

IMPACT/APPLICATIONS 

The  semi-operational  version  of  the  high  resolution  mesoscale  model  was  implemented.  This  will 
improve  5 -day  forecasting  for  aviation  and  for  support  of  Polish  troops  in  Afghanistan  and  Iraq. 

RELATED  PROJECTS 

2008  Peking  Weather  Demonstration  Project  Plans  -  We  collected  August  2005  observational  data 
and  relevant  large  scale  data  analyses  and  forecasts  in  anticipation  of  using  CO  AMPS  for  the  2008 
Olympics  WWRP  Demonstration  Project  for  China  in  support  of  the  Polish  National  Olympic  team. 
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US  GODAE  project  -  The  Global  Ocean  Data  Assimilation  Experiment  (GODAE)  provides  regular, 
complete  descriptions  of  the  state  of  the  ocean  and  the  atmosphere  in  support  of  operational 
oceanography  and  oceanographic  research.  We  use  the  GODAE  server  data  in  COAMPS  predictions. 
The  GODAE  Monterey  server  is  maintained  by  the  FNMOC  and  is  sponsored  by  the  Office  of  Naval 
Research  (ONR). 
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